The Retry That Caused the Outage It Was Trying to Prevent

A downstream service starts timing out under load. Every caller has retry logic — "resilience," in the commit message — so every failed call gets retried, usually two or three times, usually within a second or two of the original attempt. The downstream service, already struggling, now receives 3-4x its normal traffic from callers politely retrying their way through the outage. It goes from struggling to fully down. The retries didn't add resilience. They added a self-inflicted denial-of-service attack, triggered by the exact mechanism installed to prevent one.

This is the single most common way "we added retries" backfires, and it happens because retry logic, as usually written, has no concept of the system it's calling being in trouble. It just sees "this call failed" and tries again, with no memory of the last hundred calls that also failed.

What a Circuit Breaker Actually Is (Not the Library)

The circuit breaker pattern gets taught as a library integration — import the package, wrap your call, configure a threshold. But strip away the API and what it's really specifying is a personality trait: notice when you're consistently failing against the same thing, and stop trying for a while instead of continuing to hammer it. That's it. Three states — closed (normal, calls go through), open (calls fail immediately without even attempting the network call), half-open (a trickle of test calls to see if it recovered) — are just a state machine for encoding "give up temporarily" as an explicit, inspectable decision instead of an emergent property of a thousand independent retry loops that don't talk to each other.

class CircuitBreaker {
    constructor({ failureThreshold, resetTimeoutMs }) {
        this.state = "closed";
        this.failures = 0;
        this.failureThreshold = failureThreshold;
        this.resetTimeoutMs = resetTimeoutMs;
    }

    async call(fn) {
        if (this.state === "open") {
            throw new CircuitOpenError();
        }
        try {
            const result = await fn();
            this.onSuccess();
            return result;
        } catch (err) {
            this.onFailure();
            throw err;
        }
    }

    onFailure() {
        this.failures++;
        if (this.failures >= this.failureThreshold) {
            this.state = "open";
            setTimeout(() => (this.state = "half-open"), this.resetTimeoutMs);
        }
    }

    onSuccess() {
        this.failures = 0;
        this.state = "closed";
    }
}

Notice what this buys you that a retry loop alone never can: once the breaker is open, calls fail immediately, without a network round trip, without adding to the downstream service's load at all. The personality trait is "stop hurting the thing that's already hurting" — and that trait has to live somewhere in your system whether or not you ever import a library named CircuitBreaker.

Retry Logic Without This Personality Is Just Optimism

A retry with exponential backoff and jitter is a genuinely good idea for handling transient blips — a single dropped packet, a momentary GC pause. But exponential backoff alone still assumes the failure is temporary and isolated to this one call. It has no mechanism for recognizing "the last five hundred calls to this service also failed, this isn't a blip, this is an outage," because a single call's retry logic, by definition, only has visibility into itself.

That's the gap a circuit breaker closes: it's stateful across calls, in a way any individual retry attempt structurally cannot be. Backoff answers "how do I retry this one call politely." A circuit breaker answers "should any call to this service be attempted right now at all." They're not competing patterns — they answer different questions, and production resilience needs both answers, not just the one that's easier to bolt onto an existing HTTP client.

The Part the Library Can't Give You

Even with a battle-tested circuit breaker library installed correctly, teams still get burned by picking thresholds with no relationship to the actual system. A failure threshold of 50 might be reasonable for a high-traffic service where that's ten seconds of data; it might be catastrophically slow for a low-traffic internal service where 50 failures is twenty minutes of an outage you didn't notice because the breaker never tripped. The library gives you the state machine. It cannot tell you what "this service is in trouble" means for your specific traffic pattern — that judgment call is the actual engineering work, and it's the part that separates a system that fails gracefully from one that just fails at a different point in the call stack.

FAQ